A View From Egnyte Engineering

Tales from a Field Trip - An InterludeBy Krishna SankarWhile I was ready to write our continuing saga of the quest for the perfect hardware system for our cloud object store, I found some interesting blogs and discussions. Let us take a peek.

• Detour: The emphasis on system above is intentional, because optimization at component levels might not be optimal for the platform architecture as a whole and will be a topic for another day. We will discuss it in Part III, after we get past these pesky interludes. :)

Domas from Facebook writes, "Throwing more and more hardware without fine tuning for actual operational efficiency requirements is wasteful...we balance the workload so that I/O subsystem provides as efficient as possible delivery of the long tail."This was in response to Michael Stonebrakers's assertion that Facebook is trapped in 'a fate worse than death.' His arguments span three areas: scale, ops/administration and big data. Michael is somewhat right in saying that "old SQL (as he calls it) is good for nothing" and needs to be "sent to the home for retired software." The reason being that SQL databases like Oracle, DB2 and SQL Server are developed with ERP systems in mind, not the current web systems.

• Detour (again): The current trend is the NOSQL (not No SQL), which means Not Only SQL. This means that these systems augment traditional databases with scalable data stores for appropriate systems. For a quick intro you can read through my OSCON 2010 tutorial slides here.

The moral of the story, at least for us, is that one has to match the impedance of the subsystems with the platform one is building. When you are at hyper-scale, you end up innovating any platform you choose. Facebook has MySQL setup and they share the knowledge here. On a smaller sale, this is our challenge. As we scale up 10X, what are the laws that we need to break and re-learn?Epilogue: An interesting point rasied by Michael and pointed out by Domas is that "memory is the new disk." Only last week we wrote "disk is the new tape" and already people are saying memory is the new disk! Yikes, I never realized technology would move this fast. :)In the next couple of blogs, we will continue the discussion looking at the architectural pragmatics and then load testing for scalability. In fact, that may be timely as we will be working at our vendor site for the next couple of weeks, testing a few configurations across a set of hardware and service layer primitives including web layer, AES encryption, SHA-512 et al.Have a great Monday!